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This study reports the results of the formative evaluations of two computer-supported tools and the 
associated strategies for their use. Tools and strategies embedded in web-based courses can increase a 
supervisor’s involvement in helping employees transfer learning onto the workplace. Issues relating to 
characteristics of the tools and strategies as well as factors influencing their likelihood of use are identified 
via summaries of the two evaluation studies. 
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The supervisor, or the supervisory team 1 , plays an important role both in the formal and informal learning of his or 
her team members. For formal learning, the supervisor is seen as a facilitator of transfer of training, with roles 
before, during, and after the training event (Broad & Newstrom, 1992; Nelson & Dufour, 2002). For informal 
learning the supervisor can have a variety of roles such as monitoring, guiding, enabling sufficient exposure to 
learning situations, providing help and advice, or encouraging the learner to contact appropriate colleagues (Eraut, 
Alderton, Coles, & Senker, 1998). However, the supervisor is often the “weak link” in the transfer of training 
process (Robinson & Robinson, 1989). The supervisor may be reluctant to participate or may lack preparation for 
how to effectively support learning (Billett, 2002) or lack time or motivation. If the employee is involved in a course 
using Web-based technology the supervisor may lack insight into how the employee is studying because of the 
supervisor's unfamiliarity with the use of such a system. Thus when the course involves a non-traditional pedagogy, 
such as a new form of workplace learning that combines both formal and informal learning via courses that are 
centered on work-based activities learning (Bianco, Collis, Cooke, & Margaryan, 2002; Collis & Margaryan, 2003), 
the needs of the supervisor will be even stronger. 

Tools and strategies are needed to help the supervisor in his or her new role (Kessels, 1993). Using the 
terminology of human performance technology (Stolovich & Keeps, 1992), there is a performance gap between 
current and desired performance, and to close this performance gap an appropriate intervention needs to be designed 
and evaluated. If the intervention involves tools or instruments for the supervisor, the supervisor must perceive them 
as having utility (usefulness) and user-friendliness (usability) before he or she will make use of them. Pilot versions 
of the tools thus require formative evaluation to improve both their usefulness and usability. In addition, in a 
multinational corporation, when the target group of supervisors includes individuals with different management 
styles affected, among other variables, by cultural differences, the adaptability of the tools and strategies for 
different contexts needs to be considered. 

There are many sorts of tools and intervention strategies that could be used to increase the involvement of the 
supervisor in work-based learning. Some may involve incentives for the supervisor in terms of his own job- 
performance criteria. Others are under the control of the course-design team and the course facilitators. These can 
include tools such as messages or other forms of personal contact with the supervisor, informational material about 
the course directed at the supervisor, or (virtual) meetings or workshops. A learning agreement, or contract, in the 
form of a document structuring and recording a discussion and agreement between a course participant and his or 
her supervisor concerning course goals and support for course attendance is a particularly promising example of an 
intervention. Such an agreement can steer the participation of the learner towards the ways that performance change 
in the workplace is expected to be seen as a consequence of the course. Another type of intervention is particularly 
relevant for courses supported in a Web environment as it involves making supplementary material about the course 
and about how the supervisor can intervene at various times in the course for feedback to and support of the 
participant directly available to the supervisor via the Web environment. This type of intervention has the benefit of 



It should be noted that the term supervisor might be replaced by “supervisory team”. The supervisor is defined as the 
direct superior working with the team member and learner on a daily basis. However, in practice, it might be that others are 
carrying out aspects of supervision, for example when the supervisor is not an expert in the content that is being learned or “when 
teamwork is a key component of work practice” (Billet, 2002, p. 184). 
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also helping the supervisor to become familiar with the learning environment and types of learning activities in 
which the participant is involved. If a collection of different types of tools and strategies is integrated together in one 
electronic environment, this collection can be called an electronic performance support system (EPSS). 

Problem Statement 

Within the Learning & Leadership Development organization of Shell International Exploration and Production 
(LLD-SIEP) the problem of increasing supervisor involvement in learning is of particular significance because of a 
new learning strategy that focuses on work-based activities. The overall problem being investigated is: what tools 
and strategies, used in what conditions, can lead to increased supervisor involvement in technology-supported work- 
based learning? Through preliminary research (Bianco & Collis, 2003) an intervention has been identified consisting 
of an electronic performance support system (EPSS) and is in the process of development. Two particular 
components of the EPSS, a learning agreement and personalized materials added to the Web-based course 
environments supporting the participants, have been designed and are in the process of various cycles of revisions 
and formative evaluation. The problems relating to these two types of tools are: Are these types of interventions 
suitable for supervisors in the LLD-SIEP context? If they are suitable as general types, what features should they 
have in order to strengthen their likelihood of use by supervisors? Will supervisors perceive the tools as useful to 
themselves? Will they experience them as easy to use? Will it be advantageous to provide alternative forms of each 
type of tool, to anticipate differences in the supervisors themselves? The formative evaluations summarized in this 
paper address these questions. 

Theoretical Framework 

The supervisor’s performance in terms of supporting learning, as with any type of performance, can be affected by 
appropriately designed interventions. Interventions are combinations of goals, strategies, and instruments (Stolovich 
& Keeps, 1992). A distinction can be made between instructional interventions such as courses and seminars, and 
non-instructional interventions (Greenworks Consulting Training Services, 2003). The latter category includes tools 
such as electronic performance support systems (EPSSs), knowledge management (KM) tools, tools offering just-in- 
time support, tools supporting the functioning of communities of practice, and job aides and strategies related to 
corporate culture changes and process re-engineering. When the performance change being focused upon includes 
working with electronic environments, the support tools typically also have an electronic component. A major 
category of non-instructional interventions is that of electronic performance support tools. An electronic 
performance support system can also be described as “any computer software program or component that improves 
employee performance by reducing the complexity or number of steps required to perform a task, providing the 
performance information an employee needs to perform a task, or providing a decision support system that enables 
an employee to identify the action that is appropriate for a particular set of conditions” (http://www.pcd- 
innovations.com/infosite/whatepss.htm). Leighton (n.d.) indicates that an EPSS typically includes some combination 
of the components shown in Table 1: 

Table 1. Components of Electronic Performance Support (Leighton, n.d.) 



Tools 


Information Base 


Advisor 


Learning 


Word Processing, 
Spreadsheet, Database 


On-line Documents, 
Reference Materials 


Expert Advice and Coaching 


Multimedia CBT and 
Tutorials 


Templates & Forms 


Info Databases, Case 
History Data 


Context-Sensitive On-Line 
Help 


Simulations and Scenarios 



The methodology for the design and development of such an electronic intervention relates to human 
performance technology (Stolovich & Keeps, 1992) and also to software-design research more generally (Reeves, 
2000). In the design of an EPSS a number of key issues must be considered. One of these is the content of the EPSS: 
Which of the types of tools and resources indicated in Table 1 should be included? And once the choice for a type of 
component is made, how should the component be further realized? The development process of an EPSS and of its 
component tools and resources involves many iterations of pilot testing with design experts and performance 
analysts as well as the target users, in order to identify which aspects of the tool will be perceived as having the most 
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value, or utility, and which aspects will most affect the ease of use, or usability, of the tool in order to increase its 
likelihood of use in practice (Sweeney, Macquire, & Shakel, 1993). 

Research Setting 

The context for this research project is the Learning & Leadership Development organization of Shell International 
Exploration & Production B.V. (LLD-SIEP). LLD-SIEP supports learning for technical professionals within SIEP 
worldwide. At the LLD-SIEP a form of technology-supported course is being implemented based on a model of 
learning that involves different types of work-based activities, predominately carried out in the workplace with the 
support of the supervisor as well as others with relevant experience (Bianco, Collis, Cooke & Margaryan, 2002). 
Work-based learning within these courses is supported by web-based course environments that are part of the 
TeleTOP system, a blend of a course management system and learning content management system (Collis & 
Moonen, 2001). The work-based assignments involve colleagues and supervisors and are aimed at solving actual 
challenges that the learner's team is facing in practice. 

An important part of a five-year collaborative research project between the LLD-SIEP and the University of 
Twente focuses on the development of tools and strategies to support this approach to workplace learning. At the 
LLD-SIEP courses are designed by teams that include subject matter experts and designers. Tools and strategies for 
supervisor involvement are among the focuses in the design process. This portion of the overall research can be 
characterized as development research (Reeves, 2000) and has “the dual objectives of developing creative 
approaches to solving human teaching, learning, and performance problems while at the same time constructing a 
body of design principles that can guide future development efforts” (Reeves, 2000, p. 7). Figure 1 shows the model 
for part of the overall research that focuses on supervisor engagement. The dotted lines are taken as assumptions; the 
focus of the current research is on the vertical arrows relating to supervisor engagement levels and tools and 
strategies as interventions that can affect this engagement. 

Figure 1. A Model to Study Super\’isor Engagement in Technology-Supported Work-based Learning: Effect of Tools 
on Super\’isor Engagement and Effect of Supen’isor Engagement on Individual Performance. 

LEARNING AND PERFORMANCE CLIMATE 

• Supervisor engagement 

INDIVIDUAL ' '*■ TEAM LEARNING 

LEARNING AND * AND PERFORMANCE 

PERFORMANCE 

LEARNING AND PERFORMANCE DESIGN 

• Tools and strategies for work-based learning activities supported by technology 

• Tools and strategies for supervisor engagement 

Within the research represented in Figure 1, a variety of tools and strategies to involve the supervisor are being 
developed and evaluated in an iterative manner. The two examples whose evaluations are reported in this paper are 
the “Learning agreement”, an electronic template for a letter of commitment between supervisor and participant; and 
“Resources and activities embedded in the course Website”, additions to the course web-environment specifically 
designed for supervisors to encourage their participation. The formative evaluations presented in this paper are part 
of this portion of research. 

Research Questions 

This study focused on the following questions: 

1 . How are the learning agreement and course-addition tools perceived in terms of utility and usability by 
domain experts, performance-analysis experts, and tool-design experts as well as supervisors and course 
participants? 
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2. How are the learning agreement and course-addition tools perceived in terms of utility and usability by the 
LLD-SIEP course designers? 

Example 1: The Learning Agreement 

The learning agreement is one of the tools for supervisor engagement being developed within LLD-SIEP. The 
learning agreement aims at supporting participants in negotiating with their line managers for suitable conditions to 
complete the work-based course, and understanding what goals are to be achieved by participating in this course and 
how it is intended to be followed. It consists of an electronic template for a letter of commitment between supervisor 
and participant that covers goals to be achieved during the course, relation of the course to workplace needs, and 
resources available to the participants to follow the course. Participants fill in the learning agreement jointly with 
their supervisor as first activity of a course and submit the agreement in the course web-environment where it 
remains available for review and for use during the final evaluation 

The learning agreement piloted at LLD-SIEP was structured in five sections. These included: course 
participant’s details and his or her line manager’s name and email; dates during which learning agreement was valid 
and persons involved; description of a workplace situation that suggested a business need for the course and how 
successful completion of the course would impact on it; conditions under which the course participant was following 
the course, such as where and when he or she intended to study and if time would be made available for learning 
activities. It is the selection of the workplace activity or problem, which will be the focus of the course that is 
particularly critical because it is in this that the learning becomes anchored in real workplace needs. 

Pilot Test Setting 

The Learning Agreement was first designed and piloted in 2002 in a course called Health and Risk Assessment 
(Bianco, 2003). This course was designed to take practitioners involved at different levels and in different roles in 
the health- and risk-assessment process to the skill level necessary to carry out an assessment in their own 
workplaces. Participants worked both for Shell EP Operating Units worldwide and for contractor companies. The 
course consisted of seven e-modules and a number of activities that extended the concepts in the e-modules and 
were carried out in the real workplace settings of each participant. Each module consisted of a theoretical portion 
developed as an e-module and a number of self-assessment questions. After completing each e-module participants 
were required to carry out assignments that involved the application of what they had learnt in their own workplaces 
and also regularly required the involvement of the local HRA team. Results of the assignments were submitted via 
the TeleTOP system. Course participants were expected to finish the course in four weeks, but completion could 
take up to a maximum of six weeks. The course took place entirely at a distance. The learning agreement was the 
focus of the first activity of the course. The course did not involve any face-to-face component but was entirely 
carried out in the workplace, with support of the TeleTOP system. 

Method and Respondents 

This study employed qualitative methods to explore supervisors’ and participants' perceptions of the learning 
agreement as a tool to involve the supervisor in the course HRA. There were 18 participants and 15 supervisors. All 
completed a learning agreement, and all but one participant indicated his or her supervisor’s email addresses. These 
sets of participants and supervisors were invited to participate in semi-structured telephone-interviews that were 
carried out between September and October 2002. Additional documents were also analyzed to provide insight into 
the interview data. These were the course participants’ descriptions of themselves and their work uploaded in the 
TeleTOP course-management system and the learning agreements themselves. Both types of documents were 
submitted as course assignments. The uploading of participant descriptions as an assignment was meant both to 
familiarize them with the course-management system and to help them to get to know the other course participants. 

In addition to being a tool undergoing pilot testing, the learning agreement was also a key tool for data 
collection as via it participants indicated the names and emails of their supervisors. This means that only supervisors 
whose names had been indicated in the learning agreement were contacted, thus not all the supervisors whose 
employees were following the course. 

Interviews were recorded and transcribed. The supervisors’ answers were then compared to find emerging 
patterns. The same procedure occurred for participants’ answers. Then, once preliminary patterns were found, 
participants’ descriptions and learning agreements were re-analyzed to provide additional insight (Bianco, 2003). 
Results 

Response rate. Of the supervisors and participants who submitted the learning agreement, only 8 supervisors 
and 12 participants agreed to be interviewed. In the end, only 4 supervisors and 8 participants were interviewed. 
Their busy schedules and time-zone difference made it impossible to arrange a suitable time for some of the 
appointments. This response rate, 28% for the supervisors, was low. However, the course participants’ response rate 
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was also lower that expected, namely 44% (8 out of the 18 that submitted learning agreements). There appeared to 
be two reasons for this. Participants’ work descriptions in the course Web environment show that they come from 
and work all over the world, and so do their supervisors. Thus was logistically difficult to contact them and arrange 
a suitable time to interview them (phone connection, time differences). In addition, cultural differences could 
influence their perceptions of their roles and duties (Wang, Ruona, & Rojewski, 2003) and thus perhaps influenced 
the response rate. Interestingly, from the analysis of the learning agreements it emerged that participants who did not 
answer the interview request had a lower level of support from their supervisor that participants who did. Also they 
were away from their workplace for other job assignments longer. 

Participants’ and supervisors’ perceptions of the learning agreement. An interesting contrast between 
participants and supervisors emerged with respect to the utility of the learning agreements. Except for one, all 
participants thought the learning agreement was useful in getting their supervisors’ support by clarifying 
expectations and requirements for course participation and communicating to their supervisors about what they were 
doing. However, they thought the learning agreement had not helped them in getting appropriate resources (such as 
computer, or a quiet room to study) to follow the course. The supervisors’ perceptions were less enthusiastic. Two 
out of three defined the learning agreements as “just more paperwork” and could not see any utility in it. One 
suggested that the total amount of hours the course participant is expected to spend on the course should be clearly 
indicated. 

Another interesting aspect emerged with respect to the increase in workload for the participants caused both by 
the course assignments and the fact that they were expected to keep up with their already-overloaded job tasks. This 
overload seemed to be an acceptable situation in the opinion of the supervisors who did not expect any problems for 
participants to complete the course. Interestingly, this was true to a certain extent also for the participants: for five 
participants the course added to an already heavy workload, but only three of them thought they would not be able to 
meet the deadline. 

Current Status 

Taking into account the insights gained during the formative evaluation, the learning agreement is currently in 
use in all TeleTOP-supported courses now being offered to the Shell EP business. Different versions of the learning 
agreement are currently being developed as part of the ongoing research on tools and strategies. Further evaluation 
is taking place to codify and analyze the contents of the learning agreements and relate them to the impact of the 
course in the workplace. 

Example 2: Resources and Activities Embedded within the Course Website 

For the intervention piloted at LLD-SIEP and reported in this paper, resources and other additions for the supervisor 
included an introduction to the work-based learning strategy, a description of what is available for the supervisor in 
the embedded EPSS, a description of the need for a learning agreement, a set of examples of the benefits of the 
supervisor’s involvement in work-based activities, a reflection moment in the form of a questionnaire, guidelines for 
a debriefing meeting, a fill-in-form for an action plan to complete with the team member after the course, and a 
questionnaire to reflect on the team member's performance improvement and to relate the observed performance to 
what was planned for in the learning agreement. 

All these tools function as a type of embedded EPSS (Sleight, 1993). Parallel to the design of these embedded 
resources the course design also needs to call for the supervisor’s involvement via suitable activities. The embedded 
additions to the site specifically for supervisors then encourage the supervisors to know how to participate. The 
underlying strategy is that supervisors’ access to the course website is first step towards establishing a learning 
partnership. 

Resources and activities are to be included in the course roster. The roster is one of the main features of the 
TeleTOP system and consists of a matrix-type structure that displays the organization of the course and integrates 
materials, activities, and feedback (Collis & Moonen, 2001). There are many ways to make such components of an 
embedded EPSS available to supervisors. In the pilot described next, two sets of alternative ways were developed, 
presented to the target group, and evaluated. While the learning agreement pilot focused more on the utility of the 
tool, the embedded EPSS pilot focused more on the usability. 

Pilot Test 

Two alternative sets of embedded EPSS components were developed to be integrated into two otherwise 
identical course web environments. The pilot test refers here to the feedback gathered both in the development and 
evaluation of the first prototype carried out respectively in Ql, 2003 and Q2, 2003 (Tang, 2003). The methods used 
are described in the next section. 
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Method 

Development method. Resources and additions embedded within the course Website were developed using a 
rapid-prototyping approach (Reeves, 2000). A first draft of the components of the EPSS based on a preliminary task 
analysis was prepared. Feedback was collected from different stakeholders at the LLD-SIEP during individual semi- 
structured face-to-face interviews. Stakeholders included one of the directors of the LLD-SIEP, the senior learning 
advisor, two instructors, and a marketing expert. Adjustments to the design of the components followed. The 
modified new draft of the contents was later discussed during a meeting with the senior learning advisor and a 
business consultant, who agreed on the general design decisions and suggested a number of further specific design 
decisions. The general design of the components was then worked out in two different styles of language and tone of 
presentation, based on an analysis of key differences in supervisors from different regions internationally in Shell EP 
(Tang, 2003). Two versions of the embedded EPSS were then prepared, underwent an expert review, and adjusted 
once again based on the experts' comments relating to usability and aspects that differentiated one of the versions 
from the other. 

Evaluation method. Data were collected via a questionnaire that included questions about the utility and 
usability of the tools. Issues about cultural diversity were also included since the two versions of the tools were 
designed to account for differences between Western and Asian supervisors. The URL of the website where the 
tools could be viewed was included in the invitation letter for the evaluation, together with a description of the tools 
and instructions for how to fill out the questionnaire. 

Results 

Respondents. The questionnaire was sent to 25 persons. Of these, 17 answered, namely 5 LLD-SIEP learning 
designers, 1 1 students of Chinese nationality involved in the HRD Masters program at the University of Twente, and 
one learning expert from the faculty of the university. The Chinese students were included because of their HRD 
background. Besides based on their experience with managers in China they could comment on the presentation 
style and the tone of instruction. No actual Shell EP supervisors were directly involved because they had to be first 
introduced to the TeleTOP system and it was judged not convenient for them at this point in time to view such an 
early prototype. Each studied the two alternatives of embedded EPSSs and completed the evaluation questionnaire. 

Utility. The ideas of resources and activities for supervisors embedded within the course Website were generally 
well accepted, and thought to be useful and appropriate, as shown in Table 2. However, 35% of the respondents 
thought the number of activities was too much to expect of the supervisors. 

Table 2. Results with Respect to Utility (N=17). 



• Idea of using this tool to build communication with supervisors is good (88%) 

• Content is appropriate (82 %) 

• Amount of activities is too much (35%) 

• Improves supervisor’s support skills (82%) 

• Encourages supervisor’ s involvement (7 1 %) 

• Expects trainee to feel more supported by supervisors that use this tool (71%) 



Usability. Usability aspects need to be further improved, particularly in the wording of several of the sets of 
instructions. Because of reservations with respect to the clarity of wording of instructions, which were only thought 
to be attractive and persuasive by 29% of the respondents, the usability of the embedded EPSS was not considered 
to be high in its current version, as shown in Table 3. 

Table 3. Results with Respect to Usability (N=17). 



• Supervisors will like the form and layout (47%) 

• Instructions are clear (47%) 

• Instructions are attractive and persuasive (29%) 

• Instructional tone is friendly (41%) 



Current status 

After the development of the first prototype the development method for the tools and strategies has evolved. 
The researcher now works together with course designers and instructors to develop or adapt, implement, and 
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evaluate the tools as the course itself is being designed and developed and piloted. This is valuable for several 
reasons. First, both designers and instructors need to feel ownership of the tools they use, and tend not to accept 
tools that have been developed without their input. Secondly, the course design needs to include features that 
anticipate the use of the tools by the supervisors during the course so that the supervisors have a structure for 
intervening in the learning process of their team members. 

Discussion 

The pilot of the learning agreement clearly showed that attention to supervisors’ perceptions of the learning 
agreement should continue to occur in order to help evolve it into a useful tool. Supervisors’ perceptions of learning 
and other factors that influence their likelihood to become involved in learning as well as barriers to their 
involvement should be further investigated. More generally, feedback from the supervisor is of key importance in 
further developing this type of EPSS tool. Ideally, such tools should be developed jointly. There are several reasons 
for this: if a supervisor does not see the utility of a tool he or she will not use it, and more positively a supervisor can 
give suggestions to improve the tool. For example during the pilot, one supervisor suggested that the estimate of 
amount of total time required for the course should be added. The pilot of the embedded EPSSs show that offering 
options to the supervisors, in terms of tone and style of approach in the EPSS, needs to be studied further. 

Also, courses differ considerably in their design and in the conditions in which they take place. This means that 
tools need to be adaptable with each course, but also that the researcher needs to understand this complexity in order 
to develop suitable tools. An approach in which a variety of tools and versions of the tools are developed jointly 
together with course designers and instructors, piloted, and evaluated involving actual participants and supervisors is 
seen as appropriate. This will strengthen the further development of a portfolio of tools and strategies and guidelines 
for how to use them in different situations. It is this portfolio that will be integrated into the overall EPSS for 
performance support of supervisors that is to be completed in 2005. 

Contribution to HRD Knowledge 

There is not much research that focuses on the role of the supervisor when technology-supported courses focus on 
work-based activities. The current study contributes to HRD knowledge related to electronic performance support 
because it identifies tools and strategies that can be used by supervisors when employees are taking web-based 
courses and also to a new type of partnership involving educational technologists, performance analysts, course 
designers and supervisors themselves. The methodology used for the design and development work (Reeves, 2000) 
is well known in fields involving electronic performance and learning support but not often seen in HRD research. 
This is another contribution to the HRD community. 
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